Applications

Lecture 10

Author
Affiliation

Minjae Park

Auburn University
MATH 2660 - Spring 2026

Published

February 2, 2026

Markov chains

Discrete-time stochastic process

  • Suppose we observe how a quantity changes at fixed time intervals (seconds, days, weeks, years, etc.).
  • Even if we know the entire past history, the value after one time step may depend on many interacting factors.
  • Because real-world systems are complex and causal relationships are often difficult to identify, we often model the change as random (or stochastic, meaning not deterministic, but not arbitrary).
  • A mathematical model describing how a quantity evolves randomly over discrete time is called a discrete-time stochastic process.
  • Discrete-time stochastic processes are a large and important area of mathematics, with many applications.

Markov chain

  • To keep things simple, let time be indexed by \(i=0,1,2,\dots\), and denote the quantity at time \(i\) by \(X_i\).
  • Assume the quantity can take only finitely many values: \[X_i \in S=\{1,2,\dots,N\}.\]
  • The set \(S\) is called the state space, and each element is called a state.
  • For each state \(m\in S\), imagine a roulette wheel that randomly chooses the next state.
  • Once the current state is known, the next state is determined only by this roulette. (Formally, each roulette is described by a probability distribution in probability theory.)
  • The random sequence \(X_0,X_1,X_2,\dots\) is called a discrete-time finite Markov chain.

Transition matrix

  • For each state \(m\in S\), let \(\vec{p}_m=\langle p_{1m},\dots,p_{Nm} \rangle\) represent the probabilities of moving from state \(m\) to states \(i=1,\dots,N\) in one time step.
  • Each entry \(p_{nm}\in[0,1]\) is called the transition probability from state \(m\) to state \(n\).
  • Since the chain must move to some state, the probabilities in each column satisfy \[p_{1m}+p_{2m}+\cdots+p_{Nm}=1.\]
  • Collecting these vectors gives the matrix \[P=[\vec{p}_1\ \vec{p}_2\ \cdots\ \vec{p}_N],\] called the column transition matrix (or column stochastic matrix).

Example of a transition matrix

\[ P= \begin{bmatrix} 0 & \tfrac{1}{3} & \tfrac{1}{4} & \tfrac{1}{6} \\ \tfrac{1}{2} & 0 & \tfrac{1}{4} & \tfrac{1}{3} \\ \tfrac{1}{4} & \tfrac{1}{3} & 0 & \tfrac{1}{2} \\ \tfrac{1}{4} & \tfrac{1}{3} & \tfrac{1}{2} & 0 \end{bmatrix}. \]

  • The \(m\)th column describes the probabilities of moving from state \(m\) to states \(1\) through \(4\). For example, \(\vec{p}_2=\langle \tfrac{1}{3},0,\tfrac{1}{3},\tfrac{1}{3} \rangle\) means that from state \(2\), the chain moves to states \(1\), \(3\), and \(4\) with equal probability.
  • Each column sums to \(1\), meaning the chain must transition somewhere in one step.
  • Since \(p_{mm}=0\), the chain never stays in the same state in the next step.

Example (quantities)

  • Suppose there are two cities: A and B.
  • Each day:
    • \(50\%\) of planes in A fly to B (the remaining \(50\%\) stay in A),
    • \(25\%\) of planes in B fly to A (the remaining \(75\%\) stay in B).
  • Let \[\vec{n}=\langle n_A,n_B \rangle\] denote the number of planes in each city.

  • If initially there are \(100\) planes in A and \(100\) in B (that is, \(\vec{n}_0=\langle 100,100 \rangle\)), how many planes are there the next day?
  • Next-day counts:
    • In A: \(0.5\,n_A + 0.25\,n_B\),
    • In B: \(0.5\,n_A + 0.75\,n_B\).
  • In matrix form, \(\vec{n}_{\text{next}} = P\,\vec{n}_{\text{current}}\), where \[P= \begin{bmatrix} 0.5 & 0.25\\ 0.5 & 0.75 \end{bmatrix}. \]
  • This update rule does not depend on the initial numbers.
  • After \(k\) days, \[\vec{n}_k = P^k\,\vec{n}_0.\]

Example (probability)

  • Now suppose there is only one plane.
  • Each day:
    • From A: a \(50\%\) chance to move to B,
    • From B: a \(25\%\) chance to move to A.
  • If the plane starts in A, the initial probability vector is \(\vec{\pi}_0=\langle 1,0 \rangle\), meaning there is a \(100\%\) chance the plane is in A.
  • After one day, this certainty is split between cities A and B according to the transition probabilities, so \(\vec{\pi}_1=\langle 0.5,0.5 \rangle\).
  • After each day, probabilities are redistributed according to the same transition matrix: \(\vec{\pi}_{i+1}=P\vec{\pi}_i\).
  • Therefore, after \(k\) days, \(\vec{\pi}_k = P^k\,\vec{\pi}_0\).

Summary

  • A (discrete-time finite) Markov chain is a random process \(\{X_i\}_{i=0,1,2,\dots}\) that evolves over a finite state space \(S=\{1,2,\dots,N\}\).
  • At each state \(m\in S\), the next state is chosen according to the transition probabilities \(\vec{p}_m=(p_{1m},\dots,p_{Nm})\), where \(p_{nm}\) is the transition probability of moving from state \(m\) to state \(n\).
  • Collecting these probability vectors gives the transition matrix \[P=[\vec{p}_1\ \vec{p}_2\ \cdots\ \vec{p}_N],\] which completely determines the Markov chain.
  • The probability of being in each state at time \(k\) is described by a probability vector \(\vec{\pi}_k\), computed by \[\vec{\pi}_k = P^k\,\vec{\pi}_0.\]

Preview

  • If \(\vec{\pi}_k \to \vec{\pi}\) as \(k \to \infty\), the limiting probability vector \(\vec{\pi}\) is called the stationary distribution.
  • In many cases (under suitable conditions), a stationary distribution exists and is independent of the initial distribution. Intuitively, the Markov chain mixes and gradually forgets where it started.
  • Later in this course, we will learn how to compute \(P^k\) efficiently using linear algebra and understand its geometric meaning.
  • For a deeper probabilistic treatment of Markov chains, consider taking MATH 5670/5680.

Comment on convention

  • In our convention, \(p_{nm}\) represents the probability of moving from state \(m\) to state \(n\).
  • Each column sums to \(1\), so \(P\) is called a column stochastic matrix (or right stochastic matrix).
  • Many other authors define \(p_{nm}\) as the probability of moving from state \(n\) to state \(m\), so each row sums to \(1\).
  • That convention is called a row stochastic matrix (or left stochastic matrix).
  • We follow the column-stochastic convention used in the textbook, but keep in mind that references such as Wikipedia use the row-stochastic convention.

LU decomposition (optional)

This is an advanced topic and will not be included in any homework, quizzes, or exams.

Numerically solving linear equations

  • In practice, we often need to solve linear equations numerically, not just theoretically.
  • This requires algorithms that can be implemented efficiently on a computer.
  • We learned that systems of the form \(A\vec{x}=\vec{b}\) can be solved using Gauss–Jordan elimination.
  • However, Gauss–Jordan elimination is expensive if we need to solve the system many times.
  • In many applications, we reduce a matrix into simpler structured matrices so that computations are easier and more efficient.

Special matrices

  • Certain types of matrices are easier for computers to handle efficiently.
  • One example is diagonal matrices, which have nonzero entries only on the diagonal. For example, \[ \begin{bmatrix} * & 0 & 0\\ 0 & * & 0\\ 0 & 0 & * \end{bmatrix}, \qquad \begin{bmatrix} * & 0 & 0 & 0\\ 0 & * & 0 & 0\\ 0 & 0 & * & 0\\ 0 & 0 & 0 & * \end{bmatrix}. \]
  • Their geometric action is simple, and they require much less storage than full matrices.
  • Another important class consists of triangular matrices, which naturally arise from elimination algorithms.

Triangular matrices

  • An upper triangular matrix has no nonzero entries below the diagonal. For example, \[ \begin{bmatrix} * & * & *\\ 0 & * & *\\ 0 & 0 & * \end{bmatrix}, \qquad \begin{bmatrix} * & * & * & *\\ 0 & * & * & *\\ 0 & 0 & * & *\\ 0 & 0 & 0 & * \end{bmatrix}. \]
  • A lower triangular matrix has no nonzero entries above the diagonal. For example, \[ \begin{bmatrix} * & 0 & 0\\ * & * & 0\\ * & * & * \end{bmatrix}, \qquad \begin{bmatrix} * & 0 & 0 & 0\\ * & * & 0 & 0\\ * & * & * & 0\\ * & * & * & * \end{bmatrix}. \]

Backward and forward substitution

  • Linear systems with triangular matrices can be solved very efficiently:
    • backward substitution for upper triangular matrices,
    • forward substitution for lower triangular matrices.
  • As a simple example, consider solving the system \[ \left[\begin{array}{ccc|c} 1 & 0 & 0 & -1\\ 2 & 1 & 0 & 0\\ -3 & 1 & 2 & 2 \end{array}\right]. \]
  • Diagonal matrices are even more special: they are both upper and lower triangular.

Properties

  • The product of lower triangular matrices is again lower triangular.
  • The product of upper triangular matrices is again upper triangular.
  • Elementary matrices used in Gaussian elimination are lower triangular (for row scaling or row addition) or permutation matrices (for row exchange).
  • Elementary matrices used in back substitution are upper triangular. For example, \[ \begin{bmatrix} 1 & k\\ 0 & 1 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 0 & k\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} \]
  • The inverse of a triangular matrix is also triangular of the same type.

LU decomposition

  • Suppose \(A\) is an invertible \(n\times n\) matrix.
  • Assume no row exchanges are required during Gauss–Jordan elimination.
  • Then Gauss–Jordan elimination produces \[U_\ell\cdots U_1 L_k\cdots L_1 A = I_n,\] where each \(L_i\) is a lower triangular elementary matrix and each \(U_i\) is an upper triangular elementary matrix.
  • Taking inverses gives \[A = L_1^{-1}\cdots L_k^{-1} U_1^{-1}\cdots U_\ell^{-1}.\]
  • Since products of lower (upper) triangular matrices remain lower (upper) triangular, we can write \(A = LU\), where \(L\) is lower triangular and \(U\) is upper triangular.

LUP decomposition

  • If row exchanges are required during elimination, the previous argument must be modified.
  • Gauss–Jordan elimination without row exchanges produces a matrix with exactly one \(1\) in each row, but possibly not in the correct column positions.
  • A permutation matrix \(P\) rearranges coordinate directions by permuting the standard basis vectors.
  • In this case, elimination yields \[U_\ell\cdots U_1 L_k\cdots L_1 A = P.\]
  • Rearranging gives \(A = LUP\), called the LUP decomposition.

LU/LUP decomposition

  • Every invertible matrix admits an LUP decomposition \[A = LUP,\] where
    • \(L\) is a lower triangular matrix,
    • \(U\) is an upper triangular matrix, and
    • \(P\) is a permutation matrix.
  • If no row exchanges are needed during Gauss–Jordan elimination, the decomposition simplifies to an LU decomposition \[A = LU.\]
  • These decompositions are fundamental tools in numerical linear algebra.

Solving linear equations with LU/LUP decomposition

  • Suppose we want to solve \(A\vec{x}=\vec{b}\) repeatedly with the same matrix \(A\).
  • Repeating Gauss–Jordan elimination each time is inefficient.
  • Computing \(A^{-1}\) and then \(\vec{x}=A^{-1}\vec{b}\) is possible, but multiplying a large unstructured matrix to a vector is still costly.
  • Instead, we compute an LU or LUP decomposition once and reuse it for different right-hand sides.

LU decomposition and linear equations

  • Suppose we have computed \(A=LU\).
  • Then solving \(A\vec{x}=\vec{b}\) becomes \[LU\vec{x}=\vec{b}.\]
  • First solve \(L\vec{y}=\vec{b}\) by forward substitution, starting from the first equation and moving downward.
  • Then solve \(U\vec{x}=\vec{y}\) by backward substitution, starting from the last equation and moving upward.
  • Both steps are computationally very fast.

Example

  • Suppose \[ A=LU= \begin{bmatrix} 1&0&0\\ 2&1&0\\ 3&4&1 \end{bmatrix} \begin{bmatrix} 2&1&3\\ 0&1&4\\ 0&0&5 \end{bmatrix}, \] and we want to solve \[ A\vec{x}=LU\vec{x}=L\vec{y}= \begin{bmatrix} a\\ b\\ c \end{bmatrix}. \]
  • First solve \(L\vec{y}=\vec{b}\): \[ \begin{aligned} y_1 &= a,\\ y_2 &= b-2y_1,\\ y_3 &= c-3y_1-4y_2. \end{aligned} \]
  • Then solve \(U\vec{x}=\vec{y}\): \[ \begin{aligned} x_3 &= \tfrac{1}{5}y_3,\\ x_2 &= y_2-4x_3,\\ x_1 &= \tfrac12(y_1-x_2-3x_3). \end{aligned} \]

Example: electric circuit analysis

  • Electric circuit analysis is an engineering example where linear algebra naturally arises.
  • There are many other applications where matrices play a central role, including chemical reaction balancing, signal processing, control systems, differential equations, Fourier analysis, and more.
  • We will encounter additional examples throughout the course.